Problem

queue_job's jobrunner dispatches pending jobs by issuing an HTTP request to the
instance's configured URL. In a single-process setup that naturally comes back to the
same process, but in any deployment with more than one worker process behind a load
balancer or reverse proxy — for example, multiple pods or containers in a
container-orchestrated setup (Kubernetes, Docker Swarm, or similar) — that URL resolves
to whichever worker happens to be sitting behind the load balancer at that moment. Not
necessarily the worker that enqueued the job, and not necessarily one running the same
code, or even one that agrees with the database on which modules are currently
installed.

Rolling deployment: worker running stale code

Worker restarts aren't atomic across a fleet: pods/containers get replaced one at a
time, so for some window there are old and new versions running side by side, all
reachable through the same load balancer. The trigger is often the module upgrade
itself: it's a common pattern for a module's upgrade step (a post_init_hook, or a
call in a data file) to enqueue a job via with_delay() for post-processing that
shouldn't run synchronously inside the upgrade transaction — resyncing data, warming
caches, per-record post-processing, and similar. That job is created as part of the
upgrade, by the worker performing it, which is precisely what sets up the race. A
concrete timeline:

1. Worker A and worker B are both running module version 1.0, serving from the same
   database, both reachable behind the same load balancer.
2. A rolling deployment starts. Worker A restarts first, now running version 1.1, and
   runs the module upgrade (-u) against the shared database. As one of its upgrade
   steps, it enqueues a post-processing job via with_delay().
3. Worker B hasn't restarted yet — it's still running version 1.0's code, in memory,
   unaware anything changed.
4. The jobrunner's dispatch request for that job happens to be routed to worker B by
   the load balancer.
5. Worker B executes the job using version 1.0's code, against a database that version
   1.1 has already migrated.

What happens next depends entirely on what that job does — there is no single,
predictable failure mode:

• It can fail outright if the job relies on the new code but executes on the old
  worker — noisy, but at least visible.
• If the job's code path is unaffected by the schema/behavior change directly, but
  reads something version-dependent — a file bundled with the module, a piece of
  business logic that changed for the 1.1 release — it can complete "successfully"
  while producing outdated or subtly wrong results. Nothing raises, nothing logs an
  error, the job reports done. This case is the more dangerous one: it isn't caught
  by retries or monitoring, and it can go unnoticed until a symptom shows up somewhere
  else entirely, or until the next upgrade re-triggers the same job and it behaves
  differently (and correctly) the second time.

Both outcomes share the same root cause: the worker executing the job is not
guaranteed to be running the same code as the worker(s) that already updated the
database. This affects any job whose behavior is coupled to the module's own code or
bundled files, which queue_job has no way to know about or guard against on its own —
and post-processing jobs enqueued directly from an upgrade step are the most common way
to run straight into it.

A narrower, earlier window: jobs enqueued during the install itself

There's a second, related window that a version check alone can't cover, and it's the
root cause of #882. Core's own module loading
(odoo/modules/loading.py::load_module_graph()) commits a module's post_init_hook
before it commits that module's own state = 'installed' / latest_version
update. Concretely, for each module being installed/upgraded in a batch: the hook runs
and its commit fires, and only after that (sometimes as part of the next module's
commit, sometimes only at the very end of the whole request if it's the last module in
the batch) does the module get marked installed with its new latest_version.

If that hook enqueues a job — the same pattern described above — the job can already be
committed and dispatchable to any worker while the database doesn't yet record the
module as installed at all. A version-mismatch check has nothing to compare against in
that window: latest_version simply hasn't been written yet, so the job can reach a
worker that has no way to tell it's running against a database still mid-upgrade.

Core already guards against exactly this shape of problem for its own scheduled actions
(ir_cron._check_modules_state(),
odoo/addons/base/models/ir_cron.py): while any module sits in 'to install' /
'to upgrade' / 'to remove' state, cron processing is skipped for the whole
database, with a staleness escape for abandoned pending states. queue_job dispatches
over HTTP and bypasses ir_cron entirely, so it never inherited this protection.

Proposed fix

Two independent checks in RunJobController._runjob, run before a job is performed,
covering the full timeline with an atomic handover between them:

1. Worker running outdated module code

Compare each installed module's on-disk version — read fresh from this process via
get_manifest() (already lru_cached in core, so this is cheap after the first call)
— against what the database recorded as the installed version during the most recent
successful upgrade (ir.module.module.latest_version, written by
odoo/modules/loading.py every time a module is actually installed/upgraded by some
process).

A mismatch means some other worker has already upgraded this module past what the
current worker is running.

2. Install/upgrade/removal in progress anywhere in the cluster

Port of core's ir_cron._check_modules_state(): while any module's
ir_module_module.state is 'to install', 'to upgrade', or 'to remove', defer the
job — this is exactly the window described above, where a job may already be committed
and visible while the database hasn't finished recording what it belongs to. Pending
states older than a configurable timeout are treated as abandoned (e.g. a process
crashed mid-install) and ignored, with a warning logged, rather than blocking job
processing forever.

The two checks hand off cleanly: the same commit that clears the last pending module
state also publishes its latest_version, so the instant the install-in-progress check
drops, the version check has trustworthy data for any worker still running stale code.

In both cases, the job is deferred through the existing postpone/requeue path instead
of being executed — the checks run before job.perform(), so they never consume a
retry attempt and can safely wait out a rollout or install of any length. They resolve
automatically once the relevant worker is replaced/restarted or the install completes,
or sooner if a retry happens to land on a worker/state that's already current.

The version-mismatch verdict is cached briefly per database so a busy queue isn't
re-querying ir.module.module on every single job dispatch; the install-in-progress
check is intentionally not cached, since it needs to catch a job committed in the
same transaction as the pending state — a cached verdict would reopen that exact race
for the length of the cache window. Detecting an active install also busts the
version-mismatch cache, so the first job dispatched once the install completes
re-checks fresh instead of reusing a pre-install verdict.

Configuration

• queue_job.check_modules_up_to_date (default enabled) — the worker-version check.
  Set to False to disable.
• queue_job.check_install_in_progress (default enabled) — the install-in-progress
  check. Set to False to disable.
• queue_job.install_in_progress_timeout_minutes (default 60) — how long a pending
  module state is trusted before being treated as abandoned.

Tests

Added coverage for: default (not outdated / nothing pending) behavior, a forced version
mismatch, a module pending install/upgrade/removal, graceful handling of a module whose
manifest can't be read (never treated as evidence of staleness), an abandoned pending
state past the timeout, both config-parameter opt-outs, the short-lived version-check
cache (and its busting by the install-in-progress check), and all three branches of
_runjob (install-in-progress deferral, outdated-worker deferral, normal execution).

Related

Closes/addresses #882.